Skip to content

fix(memory): capture with an evaluator marks read only the messages it judged - #2836

Merged
jhoffner merged 3 commits into
mainfrom
claude/charming-gates-4duri2
Oct 7, 2026
Merged

jhoffner merged 3 commits into
mainfrom
claude/charming-gates-4duri2

Conversation

@jhoffner

@jhoffner jhoffner commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

The problem

Memory capture with an evaluator can lose a turn for good. Two turns of one session overlap. The first is judged skip, but its answer comes back late. Meanwhile the second turn's evaluator call fails. The late skip then marks the failed turn as read, so memory never observes it. The spec forbids this (FIX-1555 BR-6: "The turn stays unread and the next capture's evaluator sees it"), and the docs promise the same.

This showed up as an intermittent CI failure on a spec-only PR, #2831 (Test (3/3), run 37539012212, at 636e905). The failing test was capture-evaluator.test.ts › "a delayed skip racing an evaluator failure never marks the failed turn read", with expected '[message] I work at Acme, turn D' to contain 'turn C'.

It is a real race in read-marking, not a fragile test. The test already waits for the background capture: runAction drains side-chain work before testFlow resolves.

Root cause

The read cursor (memorySystem.lastProcessedIndex) is a position in ctx.session.items.all(). That list is not the same for every request. Each request builds its own view:

  • earlier completed requests, sorted by (ts, itemIndex), where itemIndex is counted per request;
  • then this request's own live items, at the end.

So a position taken in one request's view points at a different item in a later request's view. In the test:

  1. Turn B's capture reads its window. In B's view, B's items are at positions 0 to 4, so the boundary is 4.
  2. Turn C runs while B is still in flight, so C's view does not include B. C's evaluator call fails and nothing is marked.
  3. B's late skip sets the cursor to 4.
  4. Turn D's view merges B's and C's items by ts. On a fast runner both turns fall in one millisecond. Ties then break on the per-request itemIndex, so C's message (index 0) sorts to position 0, ahead of B's items. Cursor 4 covers it, and turn C is lost.

How I reproduced it:

  • The unmodified test passes 300/300 locally, because local item timestamps rarely collide.
  • Pinning Date so all items share one millisecond makes it fail every time, with exactly the CI message.
  • With the real clock, a skip that lands while an earlier turn is still running fails every time too. There the positional cursor gets both sides wrong: turn D re-reads the skipped turn and loses the failed one.

What this does

Capture with an evaluator now records the messages it judged by id, in a new readMessages field on the memorySystem state. It reads a message only while its id is missing from that field, wherever the message sits in the log. A capture marks read exactly what it judged.

  • Older sessions: the first capture with an evaluator carries over the messages the old watermark already covered.
  • Dropping the evaluator: capture without one clears readMessages, and the watermark decides what is read again. Re-adding an evaluator picks up from the watermark.
  • Size: an entry is dropped once it is older than every item in the completed history the capture loaded, so the field never grows past the history window. A turn still running when the capture loaded the session is not in that history, so its mark is kept. With no completed history, nothing is dropped.
  • Watermark: it still moves exactly as before. Capture without an evaluator is unchanged: it writes nothing new unless an evaluator left readMessages behind.

What's asked of you

Two decisions:

1. Does FIX-1555's spec need re-approval before this merges?

Codex flagged that adding readMessages overturns FIX-1555's "No persisted state changes". Its thread on memory-system.ts is still open.

  • My recommendation: merge without a re-gate. That line sat under Decided, not asked, which makes it an implementer note, not approved direction. The approved direction (skip is final; an evaluator error leaves the turn unread) is unchanged, and this PR is what makes it hold when turns overlap.
  • If you want it recorded: a short amendment to specs/issues/FIX-1555/DECISIONS.md in a follow-up PR from main.
  • What would change my mind: if that line was a promise to a consumer who reads the raw stored state.

2. Fix the same positional cursor on the no-evaluator path too?

Every app running memory without an evaluator still uses the positional watermark, which has the same weakness. I checked one case: a history window of 2 and 6 sequential turns. From turn 4 on, the window found no new messages. The observer saw only the raw input ([user] turn 4 instead of [message] turn 4), so the turn's other messages, such as the assistant's reply, were skipped. Overlapping turns would hit the same race as here.

  • My recommendation: a separate issue and PR. It changes the stored capture state of every memory app, so it deserves its own review and should not ride on a CI-flake fix.
  • What would change my mind: if you'd rather migrate the state shape once instead of twice.
  • Cost of waiting: the evaluator path is already correct; the gap on the default path stays as it is today.

I can file it in Linear if you want.

Parts worth reviewing closely

  • markJudgedRead and unreadMessages in packages/memory/src/memory-system-blocks.ts: the carry-over from the legacy watermark and the pruning bound (completed history only, through items.all({ includeInFlight: false })).
  • The no-evaluator memorySystemReflect branch clears readMessages only when it is set, so apps without an evaluator write exactly the state they did before.
  • memorySystemReflect's optional second parameter changed from readBoundary (returns a number) to readJudged (returns { boundary, judged }). Only memory's own capture passes it.

Links: CI failure in #2831 · FIX-1555 spec at specs/issues/FIX-1555/ · merges on its own.

How to review this

This is a bug fix on the direct route: there is no spec and no spec PR, because the hard part was the diagnosis and it happened in code. This PR is the only gate, so unlike a spec-backed change, the approach is in scope: nothing here was signed off upstream.

Most valuable here: the diagnosis chain. Does the repro actually reproduce the reported failure? Is the named cause the real one? Does the fix address that cause rather than the symptom? Does the regression test fail without the fix? Then the second path: the legacy shape, the null boundary, the concurrent case, the cancel path.

In scope to challenge: where the guard was placed, any behaviour change a user could notice, and whether this should have been a spec'd feature instead of a fix.

Everything else: verification

Tests in packages/memory/test/capture-evaluator.test.ts:

  • The CI test, kept, with a stronger assertion: it now also checks that the skipped turn stays read. The clock is pinned to one millisecond, the case that failed in CI. Without the fix it fails every time with the CI message.
  • New: "a skip never marks read a turn that was still running when it judged". Real clock; red without the fix.
  • New: "a skip never drops the mark of a turn that finished while it was judging". This is the reverse ordering Codex found. It is red when pruning uses the capture's whole view.
  • New: "an app that adds, drops and re-adds its evaluator reads each message once". This covers the legacy carry-over and the reset on the no-evaluator path.
  • New: "keeps a judged message only while the session history still holds it". This covers the pruning bound.

Each mutation was confirmed red: no carry-over, no reset, no pruning, pruning against the whole view, and marking nothing read.

Results:

  • The three race tests: 200/200 each on the final code. The first two also passed 900/900 earlier (300 repeats × 3 concurrent processes).
  • @flow-state-dev/memory: 537/537 tests pass.
  • turbo typecheck for memory and kitchen-sink is clean.
  • CI is green on fd2f873.

Changeset: @flow-state-dev/memory minor, because the exported MemorySystemState gains a required readMessages field. No docs change is needed: apps/docs/docs/memory/configuration.md already describes the corrected behaviour.

🤖 Generated with Claude Code

https://claude.ai/code/session_014pGvtq33naugpfJRZN9cDW

…t judged

The read cursor was a position in the session item log, but each request
sees its own view of that log: prior completed requests sorted by
(ts, per-request itemIndex), then its own live items. When two turns of
one session overlap, a skip's boundary, taken from its own view, lands on
a different item in a later turn's view, so it can cover a turn whose
evaluator call failed (BR-6 broken: the failed turn is never observed) or
leave a skipped turn unread. In CI the overlapping turns shared a
millisecond, the itemIndex tie-break sorted turn C's message to position
0, and turn B's late skip marked it read.

Capture with an evaluator now records the judged messages by id in a new
`readMessages` field on the memorySystem state and reads a message only
while its id is missing there. The first such capture carries over what
the legacy watermark covered; capture without an evaluator clears the
field so re-adding an evaluator resumes from the watermark; entries older
than everything in the capture's session view are dropped. The watermark
still moves as before, and capture without an evaluator is unchanged.

The flaky test now pins the clock to the one-millisecond interleaving that
failed in CI, so it fails deterministically without the fix; a second race
test covers a skip that lands while an earlier turn is still running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014pGvtq33naugpfJRZN9cDW
@vercel

vercel Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

3 Skipped Deployments
Project Deployment Actions Updated
flow-state-dev-docs Ignored Ignored Oct 6, 2026 11:50pm UTC
flow-state-dev-kitchen-sink Ignored Ignored Oct 6, 2026 11:50pm UTC
flow-state-dev-ui Ignored Ignored Preview Oct 6, 2026 11:50pm UTC

Request Review

@cursor

cursor Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: eb8b3b3a-4288-4e09-a529-5c817569031c)

The process guard requires every changeset to name its Linear issue; this
fix restores FIX-1555's BR-6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014pGvtq33naugpfJRZN9cDW
@cursor

cursor Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 34fa2164-5d59-4815-be99-7dc13d17c448)

@jhoffner
jhoffner marked this pull request as ready for review October 6, 2026 23:36
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-06T23:41:45.271406Z 202bf37 Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@cursor

cursor Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 43ab1de3-3816-467b-919e-9d32dccad09a)

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Problem

When capture uses an evaluator, “read” was expressed only as lastProcessedIndex — a position in the full session item log. Overlapping turns (same millisecond, or a slow evaluator while another turn finishes) can interleave messages in that log. A late skip could advance the watermark past a sibling turn whose evaluator failed, so that turn was never judged again.

Did this go too far?

I treated this as a /simplify pass: look for a smaller fix before accepting more state and plumbing.

Watermark-only tweaks (single-flight capture, deferring the watermark until the evaluator returns, boundary from “this turn’s items only”) do not fully repair the interleaved-log case your fake-timer / racingModel tests encode. Ephemeral sequencer-only tracking would not survive skip/reflect persisting read state across turns or evaluator on/off. Pushing read flags onto session items would be a core contract change, not a memory-package fix.

The added surface area is real (readMessages on MemorySystemState, judged on the judge input, markJudgedRead, readJudged instead of readBoundary, optional messages on readObserveWindow). It is still the minimal durable model I could find for “mark read exactly the messages this capture judged, by id.” Keeping lastProcessedIndex in sync (including clearing readMessages when capture runs without an evaluator) is migration semantics, not accidental duplication.

Simpler directions worth considering (non-blocking)

These are follow-ups, not merge blockers:

  1. DRY the watermark branch filter (idx > lastProcessedIndex && type === 'message') shared by buildObserveContext and unreadMessages.
  2. One { id, ts } Zod schema exported from memory-system.ts and reused in judgedWindowSchema (and the local ReadMessage type).
  3. Align text extraction between formatMessages and extractItemText so payload/content fallback policy cannot drift.
  4. Optional later: prune readMessages by “id still visible in current all()” instead of ts >= oldest(allItems) — might allow id-only storage if you re-prove the history-window test; I have not checked equivalence.
  5. Perf (only if profiling complains): evaluator capture already dominates; still, two items.all() calls per capture and Θ(N) scans are the scaling knobs — reuse one snapshot per capture, or incremental id sets on state.

Verification

pnpm --filter @flow-state-dev/memory test — 536 tests passed on this branch.

Verdict

Approve. The complexity matches the concurrency bug; tests document the contract (evaluator add/drop/re-add, history-window pruning, racing skip vs failed peer). Inline notes are nits / optional cleanups.

Open in Web View Automation 

Sent by Cursor Automation: Review For Best Approach

Comment thread packages/memory/src/memory-system.ts Resolved

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Snob — already tight. No POC.

I read packages/memory/src/memory-system-blocks.ts, packages/memory/src/memory-system.ts, packages/memory/test/capture-evaluator.test.ts, packages/memory/test/memory-system.test.ts, and .changeset/memory-evaluator-marks-read-by-id.md (head 202bf37c).

A rewrite would not shrink the contract:

  • unreadMessages and markJudgedRead are the fix: once readMessages is set, selection is by id; the first evaluator capture carries the watermark; pruning is by timestamp against this capture's view so a concurrent mark stays; lastProcessedIndex still advances so dropping the evaluator has a cursor.
  • formatMessages and the optional messages argument are what let the window step reuse readObserveWindow instead of copying its source and raw-input fallback.
  • toReadMessage is the snapshot both the skip path and reflect consume. Inlining it duplicates the field pick.
  • Clearing readMessages only when it is set is the no-evaluator "write the old shape" branch. Folding that into markJudgedRead moves the lines.
  • racingModel is the shared hold/fail/skip harness for the two overlap tests.

Left alone: the same positional cursor on the no-evaluator path, which this PR already split off. Not re-litigating the persisted readMessages field.

Open in Web View Automation 

Sent by Cursor Automation: Code Snob

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 202bf37c87

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread packages/memory/src/memory-system.ts Outdated
* something, and after one without an evaluator: the watermark then says
* what is read.
*/
readMessages: z.array(z.object({ id: z.string(), ts: z.number() })).nullable().default(null),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Re-gate the persisted-state change

Adding readMessages reverses the approved FIX-1555 decision that the watermark and counters retain their existing persisted shape (specs/issues/FIX-1555/DECISIONS.md:76-89). Because this is a material design change rather than an implementation detail, pause the implementation until a follow-up amendment records the new compatibility and migration trade-offs and receives renewed approval.

AGENTS.md reference: AGENTS.md:L52-L56

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this needs a re-gate, but the call belongs to the spec's owner, so I've put it to them and am leaving this thread open.

FIX-1555 listed "No persisted state changes" under Decided, not asked, as an implementer note about that change set, not as an approved direction. The approved direction (D2, BR-4, BR-6: a skip is final, and an error leaves the turn unread) is unchanged, and this PR is what makes BR-6 hold when turns overlap. The positional watermark can't express "read exactly what I judged" across overlapping turns, so keeping the old shape would mean leaving BR-6 broken. Per orchestration.md, an amendment needs renewed approval only for a materially changed direction. If the owner wants it on record anyway, the follow-up is a short amendment to specs/issues/FIX-1555/DECISIONS.md from main.


Generated by Claude Code

Pruning `readMessages` took its bound from the capture's whole session
view, which for a request is the completed history it loaded plus its own
live items. A turn that finished while this one was judging is in neither,
so when two skips overlapped, the later one dropped the earlier one's mark
and the next capture judged that turn again. The bound now comes from the
completed history alone, and nothing is dropped when there is none.

Also: the changeset is `minor` (the exported MemorySystemState gains a
required field) and one user-facing sentence; the watermark filter is one
helper; the read-message type is shared; the `source` case is commented.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014pGvtq33naugpfJRZN9cDW
@cursor

cursor Bot commented Oct 6, 2026

Copy link
Copy Markdown
Contributor

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: 4d825e3c-c079-4d4b-a697-0eef8e3d0444)

@jhoffner
jhoffner merged commit c772020 into main Oct 7, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants